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Abstract Body 
Limit 1,000 words 


Background / Context: 


Prior research and its intellectual context. 


The School Improvement Grants (SIG) program received over $3 billion through the 
American Recovery and Reinvestment Act of 2009. Through grants to states, SIG focused on 
turning around the nation’s persistently lowest-achieving schools using one of four school 
intervention models, with the aim of substantially improving student achievement. Prior 
research on SIG has documented the implementation of SIG and assessed the effectiveness of 
the program. However, rigorous evidence of the program’s impact on student outcomes has 
been limited to a few studies that focus on individual states or cities (Dee 2012, Gold et al. 
2012, LiCalsi et al. 2015). Thus, previous studies do not provide rigorous, large-scale 
evidence on whether SIG improved student outcomes. 


Purpose / Objective / Research Question: 
Description of the focus of the research. 


Did receipt of SIG funding to implement a school intervention model have an impact on the 
number of SIG-promoted practices used by low-performing schools and student outcomes? 
Consistent with the conference theme, we answered this question using a novel research 
design—a regression discontinuity design (RDD)—and a large sample of schools from many 
states to provide rigorous evidence on the effectiveness of SIG. This design exploited cutoff 
values on the continuous variables used to define the SIG eligibility tiers, comparing 
outcomes in schools that just met the eligibility cutoff to outcomes in schools that just missed 
the eligibility cutoff (while controlling for the variable used to determine tier assignment). 


Setting: 


Description of the research location. 
Low-performing schools in 22 states. 


Population / Participants / Subjects: 
Who, how many, key features or characteristics. 


We focused on the effect of SIG awards made in 2010, when over $3 billion in awards were 
made to all 50 states and DC. We selected a sample of 22 states and approximately 60 
districts that were geographically diverse, and able to support estimating impacts using a 
regression discontinuity design (RDD). To efficiently support the RDD analysis, we 
prioritized states and districts that had the largest number of schools eligible for SIG, and that 
had a high proportion of SIG-eligible schools actually receiving SIG funds to implement an 
intervention model. 


The characteristics of states in our sample did not differ significantly from states nationwide 
(see Table 1). Our sample districts had a higher percentage of non-Hispanic black students 
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and were more likely to be in an urban area than all districts in which SIG schools were 
located (see Table 2). 


Intervention / Program / Practice: 
Should including details of administration and duration. 


The treatment was defined as receiving SIG funds for implementing one of four school 
intervention models. States categorized low-performing schools into three eligibility tiers 
defined by ED. ED required each SIG-awarded school in Tier I or II to implement one of 
four school intervention models over the course of three school years (2010-2011 through 
2012-2013). Each model prescribed specific practices (see Exhibit 1). Schools in Tier III 
were permitted but not required to implement a model. Table 3 shows award amounts and the 
distribution of SIG grantees across tiers and models. 


Significance / Novelty of study: 


Description of what is missing in previous work and the contribution the study makes. 


To our knowledge, this is the first large-scale RDD that rigorously assesses the effectiveness 
of SIG using a large sample of schools from many states. The RDD tackles a host of 
methodological issues simultaneously (including multiple sites, multiple assignment 
variables, fuzziness, clustering of individuals within schools, and standard errors that take 
into account the bandwidth selection method). Although many of these issues are not new to 
the field, this study is the first to face them all at once. Because our analysis methods are 
based on simulation work and consultation with RDD experts, we think other education 
researchers will benefit from learning about how they were used to obtain the findings in this 
study. 


Research Design: 
An essential element in a SREE abstract. 


We used an RDD to estimate the impact of SIG-funded intervention models on student 
outcomes, taking advantage of RDD opportunities created by ED rules about the 
prioritization of SIG funds, which use cutoffs on continuous school-level assignment 
variables such as achievement and graduation rates. The definitions of SIG eligibility tiers 
are in Table 4 and the opportunities they create to estimate RDD impacts are in Table 5. 
Schools in Tiers I and II form the treatment group; schools in Tier III and ineligible schools 
form the comparison group. 


Data Collection and Analysis: 
Methods utilized in collecting and analyzing data. 


We collected assignment variable values for each school from states. We collected data on 
schools’ use of SIG-promoted practices using web-based surveys of school administrators in 
spring 2012 and spring 2013. We collected student-level outcomes for school years 2010- 
2011 through 2012—2013—including standardized test scores on state math and reading 
assessments, high school graduation, and college enrollment—from administrative data 
maintained by states and districts. 
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Our analytic approach included the following features (see Appendix B for more detail): 
local linear impact estimation, bandwidth selection based on the Imbens-Kalyanaraman 
(2012) method, aggregation of impacts across grades, use of covariates to increase precision, 
bootstrapped standard errors, and estimation of “fuzzy” RDD impacts. 


Findings / Results: 


An essential element in a SREE abstract. 


Findings will be available in roughly 2-3 months (December 2016 or January 2017), after 
IES releases the report focused on the impacts of SIG. 


Conclusions: 
May include recommendations and limitations based on findings. 


Conclusions will be available in roughly 2-3 months (December 2016 or January 2017), after 
IES releases the report focused on the impacts of SIG. 
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Appendices 


Not included in page count. 
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References are to be in APA version 6 format. 
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Appendix B. Analytic Approach 
Our analytic approach included the following features: 


Local linear impact estimation. We estimated impacts within a bandwidth around the cutoff 
value, adjusting for the assignment variable using a linear functional form. The coefficient on the 
assignment variable was estimated separately above and below the cutoff. 


Bandwidth selection based on the Imbens-Kalyanaraman (2012) (IK) method. We estimated 
impacts using schools with values of the assignment variable that fell within a bandwidth around 
the cutoff values. To select the bandwidth, we first standardized the outcome and assignment 
variable from each grade using the statewide mean and standard deviation for each grade. We 
centered outcome variables at their means and the assignment variable at its cutoff value. We 
divided both by their standard deviations. We then pooled the standardized variables and applied 
the Imbens-Kalyanaraman (IK) procedure to the pooled data (Imbens and Kalyanaraman 2012). 
We estimated impacts locally in each grade within the single globally selected bandwidth (that 
is, the coefficient on the assignment variable varied across grade, but we used a single bandwidth 
studywide). We chose this method because, in simulations, it produced impact estimates with 
less bias and smaller standard errors relative to alternative methods (such as cross-validation). 


Aggregation of impacts across grades. For each outcome measure, we estimated impacts 
separately for each grade—3 to 12—so that the relationship between the assignment variable and 
outcome was modeled separately across grades. We then calculated an aggregate impact that was 
a sample size weighted average of the grade-specific impacts, where the sample size was the 
number of students in study schools within the bandwidth. In calculating the weighted average of 
these impacts and the corresponding standard errors, we accounted for impact covariance due to 
overlapping samples between impacts. 


Use of covariates to increase precision. We included pre-intervention test scores and 
demographic characteristics as covariates to increase precision. We included indicator variables 
for states, districts, and RDD opportunity types to control for variation across these levels in the 
relationship between the assignment variable and the outcome. 


Bootstrapped standard errors. We estimated standard errors through residual bootstrapping to 
account for clustering of students within unique values of the assignment variable (Lee and Card 
2008), fuzziness (described below), correlations among grade impacts due to overlapping 
samples (for example, schools that met the requirement for inclusion in more than one eligibility 
tier), and variance introduced by the bandwidth selection method. 


Fuzzy RDD impacts. This study was a “fuzzy” RDD, meaning that not all schools below the 
cutoff implemented a SIG-funded intervention model, and some schools above the cutoff did. 
We calculated the impact of SIG-funded intervention models on outcomes by estimating the 
local average treatment effect (LATE). The LATE equals the RDD impact on the outcome 
divided by the RDD impact on the proportion of schools implementing a SIG-funded 
intervention model. 
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Appendix C. Tables and Figures 


Table 1. Baseline (2009-2010) characteristics of study states and all states 


Study States All States? 


Average percentage of students by 
racial/ethnic category 


White, non-Hispanic 55.3 61.8 
Black, non-Hispanic 19.5 15.8 
Hispanic 18.3 13.7 
Asian 3.8 4.6 
Other 3.1 4.1 

Percentage of studenis eligible for free or 

reduced-price lunch 48.0 45.5 
Percentage of schools that are Title | eligible 68.1 67.8 
Percentage of schools by location: 

Urban 30.0 23.3 
Suburban 25.7 22.5 
Town 14.3 16.0 
Rural 30.0 38.2 
Number of States 22 51 


Source: Common Core of Data, 2009-2010. 


Note: Data from 2008-2009 were used for states with data missing in 2009-2010. Data from 2007-2008 were 
used for states with data missing in both 2009-2010 and 2008-2009. Data from 2009-2010 were used 
whenever possible because that was the school year just before the first year of implementation of the 
American Recovery and Reinvestment Act-funded SIG models. Percentages of students and schools are 
unweighted state-level averages. There were no statistically significant differences between study states 
and all states at the 0.05 level using a two-tailed test. 


4 Includes 50 states and the District of Columbia, all of which contained schools implementing a SIG-funded 
intervention model in 2010-2011. 
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Table 2. Baseline (2009-2010) characteristics of study districts and all U.S. 
districts with schools implementing a SIG-funded model 


Districts in the United States 
with at Least One School 


Implementing a SIG-Funded 
Intervention Model 
Study Districts in 2010-2011 


Average percentage of students by 
racial/ethnic category 


White, non-Hispanic 19.5* 33.8 
Black, non-Hispanic 38.7 30.3 
Hispanic 32.0 24.4 
Asian 3.3 2.7 
Other 6.5 8.9 
Percentage of students eligible for free 

or reduced-price lunch 72.4 68.3 
Percentage of schools that are Title | 

eligible 81.4 81.3 
Percentage of districts by location: 

Urban 68.2* 39.6 
Suburban 17.3 18.7 
Town 5.7 12.0 
Rural 8.8* 29.7 
Number of Districts 60 420 


Source: Common Core of Data, 2009-2010; IES database of SIG grantees. 


Note: Data from 2008-2009 were used for districts with data missing in 2009-2010. Data from 2007-2008 were 
used for districts with data missing in both 2009-2010 and 2008-2009. Data from 2009-2010 were used 
whenever possible because that was the school year just prior to the first year of implementation of the 
American Recovery and Reinvestment Act-funded SIG models. Percentages of students and schools are 
unweighted district-level averages. The percentages of districts with at least one school implementing a 
SIG-funded model are based on schools’ planned implementation as of 2009-2010 for cohort 1 grantees, 
and only include Tier | and Tier II schools. 


*Significantly different from districts in the United States with at least one school implementing a SIG-funded 
intervention model in 2010-2011 at the 0.05 level, two-tailed test. 
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Exhibit 1. SIG Intervention Models as Described by the U.S. Department of Education SIG Guidance (2012) 
|. Turnaround Model 
A turnaround model is one in which a local education agency (LEA) must do the following: 


1) Replace the principal and grant the principal sufficient operational flexibility (including in staffing, 
calendars/time, and budgeting) to implement fully a comprehensive approach in order to substantially improve 
student achievement outcomes and increase high school graduation rates 


2) Use locally adopted competencies to measure the effectiveness of staff who can work within the turnaround 
environment to meet the needs of students: 


A. Screen all existing staff and rehire no more than 50 percent 
B. Select new staff: 


(1) Implement such strategies as financial incentives, increased opportunities for promotion and 
career growth, and more flexible work conditions that are designed to recruit, place, and retain 
staff with the skills necessary to meet the needs of the students in the turnaround school. 


(2) Provide staff with ongoing, high-quality, job-embedded professional development that is aligned 
with the school’s comprehensive instructional program and designed with school staff to ensure 
that they are equipped to facilitate effective teaching and learning and have the capacity to 
successfully implement school reform strategies. 


(3) Adopt a new governance structure, which may include, but is not limited to, requiring the school 
to report to a new “turnaround office” in the LEA or state education agency (SEA), hire a 
“turnaround leader’ who reports directly to the superintendent or chief academic officer, or enter 
into a multiyear contract with the LEA or SEA to obtain added flexibility in exchange for greater 
accountability. 


(4) Use data to identify and implement an instructional program that is research-based and vertically 
aligned from one grade to the next as well as aligned with state academic standards. 


(5) Promote the continuous use of student data (such as from formative, interim, and summative 
assessments) to inform and differentiate instruction in order to meet the academic needs of 
individual students. 


(6) Establish schedules and implement strategies that provide increased learning time. 


(7) Provide appropriate social-emotional and community-oriented services and supports for 
students. 


(U.S. Department of Education, 2012, pp. 27-28) 
ll. Restart Model 


A restart model is one in which an LEA converts a school or closes and reopens a school under a charter school 
operator, a charter management organization (CMO), or an education management organization (EMO) that has 
been selected through a rigorous review process. A restart model must enroll, within the grades it serves, any former 
student who wishes to attend the school (see C-6). 


(U.S. Department of Education, 2012, p. 31) 
lll. Closure Model 


School closure occurs when an LEA closes a school and enrolls the students who attended that school in other 
schools in the LEA that are higher achieving. These other schools should be within reasonable proximity to the closed 
school and may include, but are not limited to, charter schools or new schools for which achievement data are not yet 
available. 


(U.S. Department of Education, 2012, p. 34) 
IV. Transformation Model 
An LEA implementing a transformation model must: 
1) Replace the principal who led the school prior to commencement of the transformation model. 


2) Use rigorous, transparent, and equitable evaluation systems for teachers and principals that — 
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A. Take into account data on student growth as a significant factor as well as other factors, such as multiple 
observation-based assessments of performance and ongoing collections of professional practice 
reflective of student achievement and increased high school graduation rates. 


B. Are designed and developed with teacher and principal involvement. 


3) Identify and reward school leaders, teachers, and other staff who, in implementing this model, have increased 
student achievement and high school graduation rates and identify and remove those who, after ample 
opportunities have been provided for them to improve their professional practice, have not done so. 


4) Provide staff with ongoing, high-quality, job-embedded professional development that is aligned with the 
school’s comprehensive instructional program and designed with school staff to ensure they are equipped to 
facilitate effective teaching and learning and have the capacity to successfully implement school reform 
strategies. 


5) Implement such strategies as financial incentives, increased opportunities for promotion and career growth, 
and more flexible work conditions that are designed to recruit, place, and retain staff with the skills necessary 
to meet the needs of the students in a transformation model. 


(U.S. Department of Education, 2012, pp. 27-28) 


Source: U.S. Department of Education. “Guidance on Fiscal Year 2010 School Improvement Grants Under 
Section 1003(g) of the Elementary and Secondary Education Act of 1965.” Washington, DC: Office of 
Elementary and Secondary Education, U.S. Department of Education, 2012. Available at 
http://www2.ed.gov/programs/sif/ sigqguidance05242010.pdf. Accessed May 30, 2014. 
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Table 3. SIG funding awarded in 2010 and number of schools implementing 
each intervention model 


Yel Voto) Mi nli-yau-yaliceyamureye(=)| 


Tier Ill 
Transformation Turnaround Restart Closure strategies? 


Number of schools implementing each intervention model 


Tier | 354 138 24 8 0 524 
Tier Il 255 40 9 8 0 312 
Tier Ill 14 0 0 0 403 417 
Total 623 178 33 16 403 1,253 
Distribution of award amounts (over three years) 
10th percentile $942,892 $1,236,632 $1,187,500 $31,935 $60,190 n.a. 
50th percentile $2,100,000 $2,684,490 $2,167,965 $50,000 $300,000 n.a. 
90th percentile $5,114,190 $5,190,000 $5,490,491 $254,323 $900,405 n.a. 
Source: IES database of SIG grantees; Hurlburt et al. (2011). 
Note: The SIG awards summarized in this table are from the round of state applications due to the U.S. 


Department of Education on February 8, 2010. The award amount percentiles are based on the total award 
amount per school. 


4 Tier Ill strategies refer to all school improvement strategies used by SIG-awarded Tier III schools. Federal rules did 
not require Tier III schools to implement one of the four ED-specified school intervention models. 


n.a. = not applicable. 
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Table 4. Eligibility requirements for implementing SIG-funded intervention models’ 


Original Tier Definitions 


=>'¢ey-lare(sxem tl ke pea 


Tier | Any school receiving Title | funds for improvement, _ Title I-eligible elementary schools? that are no 
corrective action, or restructuring that falls into one higher achieving than the highest achieving school 
of the following categories: that meets the original Tier | definition and fall into 

e The lowest-achieving 5 percent of Title! one of the following categories: 
schools in improvement, corrective e The bottom 20 percent of all schools in 
action, or restructuring, or the lowest- the state based on proficiency rates 
achieving five Title | schools in e Schools that have not made AYP for two 
improvement, corrective action, or consecutive years 
restructuring in the state, whichever 
number of schools is greater 

e High schools that have had a graduation 
rate of less than 60 percent over a 
number of years 

Tier Il Any secondary school that is eligible for but does Title I-eligible secondary schools? that are 
not receive Title | funds and that falls into one of (1) no higher achieving than the highest-achieving 
the following categories: school that meets the original Tier II definition or 

e The lowest-achieving 5 percent of (2) high schools that have had a graduation rate of 
secondary schools or the lowest- less than 60 percent over a number of years and 
achieving five secondary schools in the fall into one of the following categories: 
state that are eligible for but do not e The bottom 20 percent of all schools in 
receive Title | funds, whichever number of the state based on proficiency rates 
schools is greater e Schools that have not made AYP for two 

e High schools with a graduation rate that is consecutive years 
less than 60 percent over a number of 
years 

Tier Ill | Schools receiving Title | funds for improvement, Title I-eligible schools® that do not meet the 
corrective action, or restructuring that are not in requirements to be in Tier | or Tier II and that fall 
Tier | into the following categories: 

e The bottom 20 percent of all schools in 
the state based on proficiency rates 
e Schools that have not made AYP for two 
years 
Source: U.S. Department of Education. 


4 The original tier definitions were published in the Federal Register on December 10, 2009. The expanded tier 
definitions were published in the Appropriations Act on December 16, 2009. 


» Title I-eligible schools include all schools eligible to receive Title | funds, including both those that do and do not 
actually receive the funds. 


AYP = adequate yearly progress. 
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Table 5. Opportunities to conduct an RDD based on SIG eligibility tier definitions® 


Opportunity 


TaiCcaVclavece)smeigelel e) 


Comparison Group 


Assignment Variable 


1 Original tier | elementary Original tier IIl elementary Achievement 

2 Original tier | secondary Original tier II secondary Achievement 

3 Original tier | secondary Original tier II secondary Graduation rate 

4 Original tier II secondary Original tier Il secondary, Achievement 
but above the cutoff® 

5 Original tier Il secondary Original tier Il secondary, Graduation rate 
but above the cutoff¢ 

6 Expanded tier | elementary Expanded tier Ill elementary Achievement 

- Expanded tier Il secondary Expanded tier IIl secondary Achievement 

8 Expanded tier Il secondary Expanded tier Ill secondary Graduation rate 


Source: State administrative records. 


4 The original tiers were based on the definitions published in the Federal Register on December 10, 2009. The 
expanded tiers were based on the definitions published in the Appropriations Act on December 16, 2009. 


> Schools that were eligible for but did not receive Title | funding and were above the 5 percent achievement cutoff. 
° Schools that were eligible for but did not receive Title | funding and were above the 60 percent graduation rate cutoff. 
RDD = regression discontinuity design. 
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